Image processing apparatus and information processing method for improving resolution of direction exhibiting degraded resolution

ABSTRACT

An image processing apparatus includes an acquisition unit configured to acquire a first finite spatial filter having image resolution anisotropy, and a calculation unit configured to compute a second spatial filter by convolving a finite filter with respect to the first spatial filter, the finite filter having a sum of elements being 0 and at least two of the elements being non-0.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based upon and claims the benefit of priorityof the prior Japanese Patent Application No. 2012-141394 filed on Jun.22, 2012, the entire contents of which are incorporated herein byreference.

FIELD

The disclosures herein generally relate to an image processing apparatusand an information processing method.

BACKGROUND

In images taken by a digital camera or the like, deterioratedresolutions may frequently be observed in peripheral parts of suchimages. The resolution in the peripheral parts of the image taken by thedigital camera or the like may be likely to be deteriorated since thefield angle relies on the aberrations or the aperture sizes of a lensoptical system. One of the factors of such deteriorated resolution maybe mechanical vignetting. Mechanical vignetting may be observed in apart of the aperture having a large field angle in a radial direction.This part of the aperture forms an elliptic shape, thereby inducingimage blurring. Thus, the resolution in the radial direction maydeteriorate.

In order to overcome such deterioration of the resolution, JapaneseLaid-open Patent Publication No. 2012-23498 (hereinafter referred to as“Patent Document 1”), for example, discloses an image correctiontechnology to correct an image to adjust a change in PSF (point spreadfunction) due to an angle of incident light by filtering the image to beprocessed with respect to different positions utilizing different filterdata.

RELATED-ART DOCUMENT Patent Document

-   Patent Document 1: Japanese Laid-open Patent Publication No.    2012-23498

Note that resolution has a property that differs according to differentdirections. Such a property of the resolution is hereinafter called“resolution anisotropy”. For example, the resolution in a radialdirection may differ from the resolution in a circumferential direction.

Hence, although the image blurring is corrected by performing differentfiltering according to the positions of the image as disclosed in therelated art technology, the related art technology may not be able toimprove the resolution anisotropy.

On the other hand, in order to implement a filtering function inhardware of the digital camera or the like, the filtering function isconvolved with a finite spatial filter so as to correct the imageblurring. This is because a large number of operations may be requiredfor an operation in a frequency domain, such as Fourier transform.However, when the spatial filter having anisotropy is limited to afinite number of elements, deterioration in high frequency componentsmay be acquired in different directions to thereby induce a directionalmoiré pattern.

SUMMARY

According to one aspect of the present invention, an image processingapparatus may include an acquisition unit configured to acquire a firstfinite spatial filter having image resolution anisotropy; and acalculation unit configured to compute a second spatial filter byconvolving a finite filter with respect to the first spatial filter, thefinite filter having a sum of elements being 0 and at least two of theelements being non-0.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of an optical system;

FIG. 2 is a diagram illustrating an example of an aperture according todifferent positions of an image;

FIG. 3 is a diagram illustrating image blurring directions according tothe different positions of the image;

FIG. 4 is a diagram illustrating resolution when a wedge chart isemployed;

FIG. 5 is a diagram illustrating a result of the resolution measured bythe wedge chart;

FIG. 6 is a diagram illustrating deterioration observed at differentimage-captured positions;

FIG. 7 is a diagram illustrating an example of the wedge chart at an endpart of the captured image;

FIG. 8 is a diagram illustrating an example of a resolution analysisresult of the wedge chart illustrated in FIG. 7;

FIG. 9 is a diagram illustrating an example of a central part of thecaptured image of the wedge chart;

FIG. 10 is a diagram illustrating an example of a resolution analysisresult of the wedge chart illustrated in FIG. 9;

FIG. 11 is a diagram illustrating an example of characteristics when animage-blurring function undergoes Fourier transform;

FIG. 12 is a diagram illustrating an inverse number of K(ω);

FIG. 13 is a diagram illustrating an inverse filter having a constantvalue added to the denominator;

FIG. 14 is a diagram illustrating an inverse filter that drops gain witha higher frequency;

FIG. 15 is a diagram illustrating an example of an elliptic PSF (pointspread function);

FIG. 16 is a diagram illustrating an example of characteristics when theelliptic image-blurring function undergoes Fourier transform;

FIG. 17 is a diagram illustrating an inverse number of K(ω,θ);

FIG. 18 is a diagram illustrating an inverse filter having a constantvalue added to a denominator;

FIG. 19 is a diagram illustrating an inverse filter that drops gain witha higher frequency;

FIG. 20 is a diagram illustrating an example of an inverse filter forimproving resolution anisotropy;

FIG. 21 is a block diagram illustrating a schematic configurationexample of an image pickup apparatus including a spatial filtergenerator;

FIG. 22 is a diagram illustrating rotations;

FIG. 23 is a diagram illustrating an example of a two-dimensionalspatial frequency distribution of an inverse filter K_(inv);

FIG. 24 is a diagram illustrating an example including distributionsaccording to spatial frequency directions of the inverse filter K_(inv);

FIG. 25 is a block diagram illustrating a functional example of acoefficient analysis part;

FIG. 26 is a diagram illustrating a computational procedure of the PSF;

FIG. 27 is a diagram illustrating an example of an image having 12imaged wedge charts;

FIG. 28 is a diagram illustrating an example of a spatial filter table;

FIG. 29 is a diagram illustrating an example of a spatial filter;

FIG. 30 is a diagram illustrating intensity of a finite spatial filter;

FIG. 31 is a diagram illustrating an example of a moiré patterngenerated in a corrected image;

FIG. 32 is a diagram illustrating a resolution analysis result of theimage corrected by the finite spatial filter;

FIG. 33 is a diagram illustrating a resolution analysis result of theimage corrected by suppressing the moiré pattern;

FIG. 34 is a diagram illustrating drastic luminance changes;

FIG. 35 is a diagram illustrating luminance changes after the image ispassed through a high-pass filter;

FIG. 36 is a block diagram illustrating a schematic configurationexample of an image pickup apparatus including an image processingapparatus according to a first embodiment;

FIG. 37 is a block diagram illustrating a schematic configurationexample including a filter controller and a filter processor accordingto the first embodiment;

FIG. 38 is a diagram illustrating intensity of each of pixels in afinite spatial filter F₉;

FIG. 39 is a diagram illustrating intensity of each of pixels in thefinite spatial filter F₉;

FIG. 40 is a diagram illustrating an image (a concept) of a correctingprocess according to the first embodiment;

FIG. 41 is a diagram illustrating linear interpolation of a focused-onpixel;

FIG. 42 is a diagram illustrating a resolution analysis result beforethe image correction;

FIG. 43 is a diagram illustrating a resolution analysis result after theimage correction according to the first embodiment;

FIG. 44 is a flowchart illustrating an example of a filter computationprocess according to the first embodiment;

FIG. 45 is a flowchart illustrating an example of a filter processaccording to the first embodiment;

FIG. 46 is a block diagram illustrating a schematic configurationexample including a filter controller and a filter processor accordingto a second embodiment;

FIG. 47 is a diagram illustrating intensity of each of pixels in afinite spatial filter F′₉;

FIG. 48 is a diagram illustrating an image (a concept) of a correctingprocess according to the second embodiment;

FIG. 49 is a flowchart illustrating an example of a filter computationprocess according to the second embodiment;

FIG. 50 is a flowchart illustrating an example of a filter processaccording to the second embodiment;

FIG. 51 is a block diagram illustrating a schematic configurationexample of an image processing apparatus according to a thirdembodiment;

FIG. 52 is a diagram illustrating an example of a chart (a firstexample);

FIG. 53 is a diagram illustrating an example of a chart (a secondexample); and

FIG. 54 is a diagram illustrating an example of a chart (a thirdexample).

DESCRIPTION OF EMBODIMENTS

Initially, one of the factors that may deteriorate resolution isdescribed with reference to FIGS. 1 through 3. FIG. 1 is a diagramillustrating one example of an optical system. FIG. 2 is a diagramillustrating an example of an aperture according to different positionsof an image. When the optical system illustrated in FIG. 1 is employed,the aperture in the center of an optical axis is circular. However, theaperture with a large field angle may induce mechanical vignetting asillustrated in FIG. 2. The aperture may be elliptic according topositions of the image, as illustrated in FIG. 2.

FIG. 3 is a diagram illustrating image blurring directions according tothe different positions of the image. When the optical systemillustrated in FIG. 1 is employed, and the aperture is narrowed down,the resolution may deteriorate. As a result, the image blurring may belikely to be increased in a radial direction as illustrated in FIG. 3.

Next, a resolution analysis conducted by inventors of the presentapplication is described. The deterioration of the resolution isanalyzed in detail by imaging a Siemens star (hereinafter also called a“chart”), which consists of a pattern of lines radiating from a commoncenter and being distributed (spread) toward the edges of the lines.

FIG. 4 is a diagram illustrating resolution when employing a wedgechart. In an example illustrated in FIG. 4, in order to measure theresolution in a direction indicated by an arrow, plural data may need tobe acquired in a direction perpendicular to the direction indicated bythe arrow. In a wedge-shaped Simens star illustrated in FIG. 4, thelines become narrower the closer to the common center of their edges.Therefore, the number of lines per pixel unit may increase as the linesget closer to the common center. The center of the wedge-shaped Simensstar represents a high frequency component. Further, the intensity ofthe luminance value may decrease as the lines get closer to the commoncenter of their edges.

As illustrated in FIG. 4, directional resolution or a MTF (modulationtransfer function) may be analyzed by employing a subject such as thewedge-shaped chart radiating in all directions.

FIG. 5 is a diagram illustrating a measured result of the resolutionanalyzed by the wedge chart. More specifically, FIG. 5 is a graphillustrating the resolution measured in the direction illustrated inFIG. 4. In FIG. 5, a vertical axis indicates the luminance value and ahorizontal axis indicates the number of lines (LP: Line Pair) per pixel.This analysis illustrates a process (MTF) of the intensity thatdecreases the closer to the center and that deteriorates the higher thefrequency component (i.e., a rightward direction in the horizontalaxis).

FIG. 6 is a diagram illustrating the deterioration of the intensityaccording to positions of an image. In an example illustrated in FIG. 6,when imaging aligned plural Simens stars, the resolution of an end partmay be analyzed with respect to the center of the image-capturedpositions. In the example illustrated in FIG. 6, Dx indicates acircumferential direction and Dy indicates a radial direction. The abovedefinitions of Dx and Dy are also applied to subsequent drawings.

The analysis result illustrated in FIG. 6 indicates that a peripheralpart of the image including its end parts not only exhibitsdeterioration of the resolution but also exhibits resolution anisotropy.In comparing the resolutions of the aligned Simens stars, angledependency may little be observed in the center of the image; however,angle dependency may be observed in the end parts of the image.

FIG. 7 is a diagram illustrating an example of the wedge chart at theend part of the captured image. The resolution of the wedge chartillustrated in FIG. 7 is analyzed in an orthogonal direction of the Dxdirection (i.e., the radial direction) and in an orthogonal direction ofthe Dy direction (i.e., the circumferential direction).

FIG. 8 is a diagram illustrating an example of a resolution analysisresult of the wedge chart illustrated in FIG. 7. As illustrated in FIG.8, the resolution in the radial direction exhibits deterioration greaterthan the resolution in the circumferential direction. This indicatesthat there is resolution anisotropy in the end parts of the image, andit is possible to measure the resolution quantitatively.

FIG. 9 is a diagram illustrating an example of the wedge chart at thecentral part of the captured image. The resolution of the wedge chartillustrated in FIG. 7 is analyzed in an orthogonal direction of the Dxdirection (i.e., the radial direction) and in an orthogonal direction ofthe Dy direction (i.e., the circumferential direction).

FIG. 10 is a diagram illustrating an example of a resolution analysisresult of the wedge chart illustrated in FIG. 9. As illustrated in FIG.10, there is no such difference between the resolution in the radialdirection and the resolution in the circumferential direction. Thus, theresolution anisotropy will not be observed in the central part of theimage.

Note that in order to correct image blurring including theabove-described resolution deterioration, there exists a correctingmethod by utilizing the point spread function (PSF). The PSF is afunction representing image blurring, for example. In the followingdescription, such a function representing image blurring is also calledan “image-blurring function”.

When an original image is represented by x, and the PSF is representedby k, a blurred image y is an image obtained by convolving x and k,which is represented by the following formula (1).y=k⊗x   (1)

⊗: Convolution

The actual image includes noise n; however, the noise n is omitted fromthe description for convenience of illustration.

The formula (2) is obtained by applying Fourier transform to the formula(1).Y(ω)=K(ω)X(ω)   (2)ω: spatial frequencyNext, an inverse filter K_(inv) may simply be obtained by an inversenumber of K.K_(inv)(ω)=1/K(ω)   (3)Thus, the Fourier transform X(ω) of the original image is computed bythe following formula (4), and the inverse Fourier transform is appliedto the computed result of the formula (4) so as to obtain the originalimage.X(ω)=K_(inv)(ω)Y(ω)   (4)

As described above, when the Fourier transform is applied to the PSF tocompute an inverse filter function (hereinafter simply called the“inverse filter”) based on the inverse number, a division is performedin the spatial frequency, which may induce a division by 0 in the highfrequency domain. The division by 0 indicates a division is performed by0 or a value close to 0. When the high frequency is close to 0, aninverse number may be too large, which may intensify the noise of thehigh frequency.

FIG. 11 is a diagram illustrating an example of characteristics when theimage-blurring function undergoes the Fourier transform. FIG. 11illustrates K(ω), which approximates to 0 in the high frequency.

FIG. 12 is a diagram illustrating an inverse number of K(ω). FIG. 12illustrates K_(inv) represented by the above formula (3) and having adenominator of approximately 0 in the high frequency. Accordingly, noisein the high frequency may be increased.

Hence, in order to reduce such noise in the high frequency, a correctionfactor is introduced in the denominator of K_(inv) to prevent the noisefrom being intensified.K_(inv)(ω)=1/(K(ω)+λ)  (5)K_(inv)(ω)=K(ω)/(K(ω)K(ω)+λλ)  (6)

K(ω): Complex conjugate

Since the inverse filter is formed of a complex number, the inversefilter is represented by a complex conjugate.

FIG. 13 is a diagram illustrating an inverse filter having a constantvalue added to the denominator. FIG. 14 is a diagram illustrating aninverse filter that drops gain with a higher frequency. As illustratedin FIGS. 13 and 14, the inverse filter lowers the noise by addingweights for each of the frequency components.

Note that a case in which the PSF (point spread function) is elliptic isconsidered below. FIG. 15 is a diagram illustrating an example of anelliptic PSF. In the example illustrated in FIG. 15, the resolution inthe Dy direction is lower than the resolution in the Dx direction. Thatis, the resolution in the Dy direction exhibits deterioration greaterthan the resolution in the Dx direction.

The elliptic PSF is represented by k(r, θ). r represents a radius and θrepresents a direction. The elliptic PSF may be represented as afunction of the radius r and the direction θ. When the Fourier transformis applied to the elliptic PSF, the equation K(ω, θ)=fk(r, θ) isobtained. f represents the Fourier transform. K(ω, θ) obtained after theapplication of Fourier transform is a function of a spatial frequency ωand a direction θ.

FIG. 16 is a diagram illustrating an example of characteristics when theelliptic image-blurring function undergoes the Fourier transform. Theexample of FIG. 16 employs the image-blurring function illustrated inFIG. 15 such that the characteristics may differ according to adirection θ. As illustrated in FIG. 16, the characteristics in the Dxdirection differ from the characteristics in the Dy direction having thedeteriorated resolution.

FIG. 17 is a diagram illustrating an inverse number of K(ω, θ). FIG. 17illustrates K_(inv) represented by the above formula (7) and having adenominator of approximately 0 in the high frequency. Accordingly, noisein the high frequency may be increased.K_(inv)(ω,θ)=1/K(ω,θ)  (7)

Hence, in order to reduce such noise in the high frequency, a correctionfactor is introduced in the denominator of K_(inv) so as to prevent thenoise from being intensified. The following formula (8) represents aninverse filter to reduce the noise in the high frequency.K_(inv)(ω,θ)=K(ω,θ)/(K(ω,θ)K(ω,θ)+λλ)  (8)

FIG. 18 is a diagram illustrating an inverse filter having a constantvalue added to the denominator. FIG. 19 is a diagram illustrating aninverse filter that drops gain with a higher frequency. As illustratedin FIGS. 18 and 19, the inverse filter lowers the noise by addingweights for each of the frequency components.

The elliptic image-blurring function (e.g., PSF) may also reduce thenoise in the high frequency based on the corrected weights (λ). However,with this correction based on weights (λ), it may be difficult toimprove the correction in a direction having the deteriorated resolution(e.g., the Dy direction). Hence, it may be difficult to improve theresolution anisotropy by simply adding the weights (λ). However, it maybe possible to improve the resolution anisotropy by appropriatelyadjusting the weight function according to the different directions.

FIG. 20 is a diagram illustrating an example of an inverse filter forimproving the resolution anisotropy. In the example illustrated in FIG.20, the gain may be likely to be dropped with the higher frequency;however, the degraded resolution in the Dy direction may significantlybe improved.

Next, the generation of a spatial filter for improving resolutionanisotropy is described. As illustrated in FIG. 20, an example of amethod of dropping the gain may be described with reference to a deviceillustrated blow. However, the method is not limited to the exampleillustrated in FIG. 20. For example, the method of adding the constantvalue as illustrated in FIG. 13 may also improve the resolutionanisotropy. In this case, the constant value may be added such that theresolution in the Dy direction is intensified higher than the resolutionin the Dx direction.

Spatial Filter Generator

Initially, a spatial filter generator for improving resolutionanisotropy is described.

Configuration

FIG. 21 is a block diagram illustrating a schematic configurationexample of an image pickup apparatus including a spatial filtergenerator. The image pickup apparatus illustrated in FIG. 21 includes anoptical system 1, an image pickup device 2, an AFE (analog front end) 3,an image processor 4, a post-processor 5, a drive controller 6, acontroller 7, an image memory 8, a display part 9, and a coefficientanalysis part 10.

The optical system 1 is configured to converge light on an image surfacefrom a subject K having radial patterns. For example, the optical system1 includes lenses 11a, 11b, 11c, and a diaphragm 12. The lenses 11a,11b, 11c, and the diaphragm 12 of the optical system 1 converge lightfrom the subject K to form an image of the subject K onto an imagepickup surface of the image pickup device 2. The drive controller 6 maybe able to control respective positions of the lenses 11a, 11b, and 11c,and the levels (degrees) of an aperture stop of the diaphragm 12. Notethat the configuration of the optical system 1 is not limited to aspecific configuration.

The image pickup device 2 is configured to convert the light convergedby the optical system 1 from the subject K into electrical signals(analog signals). The image pickup device 2 may, for example, include atwo-dimensional image pickup device such as a CCD or CMOS. Such atwo-dimensional image pickup device is configured to convert an image ofthe subject into electrical signals and output the converted electricalsignals to the AFE 3.

The AFE 3 is configured to convert the analog signals of the capturedimage into digital signals. The AFE 3 may, for example, include an AD(analog-to-digital) converter 31 and a timing generator 32. The timinggenerator 32 is configured to generate a timing pulse utilized fordriving the image pickup device 2 based on a control signal from thecontroller 7 and output the generated timing pulse to the image pickupdevice 2 and the AD converter 31.

The image processor 4 is configured to store an image formed of thedigital signals and perform a predetermined image process on the imageof the digital signals. The image processor 4 may, for example, includean RAW memory 41 configured to record the converted image (i.e., RAWimage) that is converted by the AD converter 31 into the digitalsignals. The image processor 4 may perform the predetermine process onthe RAW image. The image on which the predetermined process is performedis recorded in the image memory 8.

The post-processor 5 is configured to perform a further necessaryprocess on the image on which the predetermined process has beenperformed to generate a display image. The post-processor 5 may, forexample, retrieve from the image memory 8 the image on which thepredetermined process has been performed, and perform the necessaryprocess on the retrieved image to generate the display image that isoutput to the display part 9.

The image memory 8 is configured to store the image obtained after thepredetermined process has been performed on the image. The display part9 may, for example, include a VRAM (i.e., a video RAM) configured torecord an image and a display configured to output the image recorded onthe VRAM. Note that the image pickup apparatus does not necessarilyinclude a display function, and may include a recording part (e.g., theVRAM or the like) configured to record an image for display instead ofthe display part 9.

The drive controller 6 is configured to control the optical system 1.The controller 7 is configured to control the AFE 3, and thepost-processor 5.

The coefficient analysis part 10 is configured to analyze resolution indifferent directions at different image-captured positions from theimage of a chart, and determine appropriate filter data for improvingthe resolution anisotropy based on the analyzed result. The details ofthe coefficient analysis part 10 will be described later.

The filter data may, for example, be a group of parameters necessary forimage correcting filtering such as a deconvolution kernel. Thedeconvolution kernel may specifically be expressed by a circular or anelliptic subject image distribution domain according to the PSF and datarepresenting weight each of the pixels in the domain (such data arecalled a “deconvolution distribution”).

Inverse Filter

Note that an inverse filter used in the embodiment is described. In thefollowing, a description is given of a procedure to compute the inversefilter configured to adjust the direction exhibiting the degradedresolution to improve the resolution anisotropy. The inverse filter maysimply be called a “filter”.

As illustrated in the formula (1), the original image x, the PSF k, andthe blurred image y are considered. In computing the original image x,an image approximating the original image x may be obtained when thefollowing formula (9) is minimum as an inverse problem.∥y−k⊗x∥²  (9)

Normally, some kind of a regularization term is applied for solving theinverse problem. Thus, the inverse problem may be solved by the formula(10) having the additional regularization term.∥y−k⊗x∥²+ϵE(x)  (10)

The above problem may require directivity. Hence, differential terms ofa transverse (horizontal) direction (x direction), and a longitudinal(vertical) direction (y direction) are added together with theregularization term in the problem.∥y−k⊗x∥²+ϵ{∥d_(m)⊗x∥²+∥d_(m)⊗x∥²}  (11)ϵ: weighting factordm, dn: differential filters in matrix directions

$d_{m} = \begin{bmatrix}{- 1} & 1\end{bmatrix}$ $d_{n} = \begin{bmatrix}{- 1} \\1\end{bmatrix}$

To minimize the formula (11), a result obtained by differentiatingpartially the formula (11) by x may simply be set to 0, and thefollowing formula (12) is obtained when X(ω) is solved by theapplication of the Fourier transform.

$\begin{matrix}{{X(\omega)} = \frac{{K(\omega)}{Y(\omega)}}{{K(\omega)}^{2} + {ɛ\left\{ {{D_{m}(\omega)}^{2} + {D_{n}(\omega)}^{2}} \right\}}}} & (12)\end{matrix}$X(ω), Y(ω), K(ω), Dm(ω), and Dn(ω) represent Fourier transformed x, y,dm, and dn, respectively.

The inverse filter K_(inv)(ω) in the frequency domain satisfies thefollowing formula (13).X(ω)=K_(inv)(ω)Y(ω)   (13)

Accordingly, the inverse filter K_(inv)(ω) satisfies the followingformula (14).

$\begin{matrix}{{K_{inv}(\omega)} = \frac{K(\omega)}{{K(\omega)}^{2} + {ɛ\left\{ {{D_{m}(\omega)}^{2} + {D_{n}(\omega)}^{2}} \right\}}}} & (14)\end{matrix}$

When the formula (14) employs the complex conjugates, the followingformula (15) is obtained.

$\begin{matrix}{{{K_{inv}(\omega)} = \frac{\overset{\_}{K(\omega)}}{{{K(\omega)}\overset{\_}{K(\omega)}} + {ɛ\left\{ {{{D_{m}(\omega)}\overset{\_}{D_{m}(\omega)}} + {{D_{n}(\omega)}\overset{\_}{D_{n}(\omega)}}} \right\}}}}{\overset{\_}{K(\omega)}\text{:}\mspace{14mu}{Complex}\mspace{14mu}{conjugate}}} & (15)\end{matrix}$

In this embodiment, in order to adjust the direction exhibiting thedegraded resolution, the axis of a differential coefficient is rotatedinto an angle θ direction by utilizing a rotation matrix.D_(x)(ω,θ)=D_(m)(ω)cos θ−D_(n)(ω)sin θ  (16)D_(y)(ω,θ)=D_(m)(ω)sin θ+D_(n)(ω)cos θ  (17)That is, the directivity is provided by utilizing the rotation matrix.

FIG. 22 is a diagram illustrating rotations. In the example illustratedin FIG. 22, the Dn direction is rotated by θ such that the Dn directionbecomes the Dy direction, and the Dm direction is rotated by θ such thatthe Dm direction becomes the Dx direction.

Note that the ellipse PSF is represented by k(r, θ), and the ellipse PSFobtained after the application of the Fourier transform is representedby K(ω, θ)=fk(r, θ). When the formulas (16) and (17), and K(ω, θ) aresubstituted into the formula (15) as well as setting the weights raccording to the different directions, the following formula (18) isobtained.

$\begin{matrix}{{K_{inv}\left( {\omega,\theta} \right)} = \frac{\overset{\_}{K\left( {\omega,\theta} \right)}}{\begin{matrix}{{{K\left( {\omega,\theta} \right)}\overset{\_}{K\left( {\omega,\theta} \right)}} +} \\{ɛ\left\{ {{{D_{X}\left( {\omega,\theta} \right)}\overset{\_}{D_{X}\left( {\omega,\theta} \right)}} + {{{\gamma D}_{Y}\left( {\omega,\theta} \right)}\overset{\_}{D_{Y}\left( {\omega,\theta} \right)}}} \right\}}\end{matrix}}} & (18)\end{matrix}$γ: weighting factor according to directions of the inverse filterϵ: entire weighting factor

The above formula (18) may be able to implement the directionalweighting adjustment of the inverse filter utilized in each of theembodiments. For example, the coefficient analysis part 10 adjusts theweight γ of the direction (Dy direction) exhibiting the degradedresolution. Hence, it may be possible to improve the directionexhibiting the degraded resolution by reducing the weighting factor γ.

FIG. 23 is a diagram illustrating an example of a two-dimensionalspatial frequency distribution of an inverse filter K_(inv). The exampleillustrated in FIG. 23 indicates a case where γ=1/300. The weightingfactors γ and ϵ (especially γ) are determined such that the degradedresolution in the Dy direction is improved greater than the degradedresolution in the Dx direction.

FIG. 24 is a diagram illustrating an example of a spatial frequencydirectional distribution of the inverse filter K_(inv). As may be clearfrom FIG. 24, the degraded resolution in the Dy direction may exhibitthe improvement significantly higher than that in the Dx direction byreducing the weighting factor of the Dy direction. Further, Dy′ in FIG.24 indicates a distribution in a case where γ=1. Hence, anisotropy mayalso be given to the weighting factors by which the filter parameters ofthe inverse filter are multiplied according to the embodiment.

Coefficient Analysis Part

Next, the coefficient analysis part 10 is illustrated. The coefficientanalysis part 10 is configured to determine a spatial filter forimproving resolution anisotropy.

FIG. 25 is a block diagram illustrating a functional example of thecoefficient analysis part 10. The coefficient analysis part 10illustrated in FIG. 25 includes a resolution analysis part 101, and adetermination part 102. The coefficient analysis part 10 is configuredto perform the coefficient analysis based on the image including thechart image. In the following example, the coefficient analysis isperformed on a wedge-form image captured in the upper left position.

The resolution analysis part 101 is configured to analyze deterioratedor degraded resolution of the image including a subject having radialpattern at least in two directions. The technique of the analysis mayemploy a method described in FIG. 4 or FIG. 5.

The resolution analysis part 101 detects the number of lines per pixelin the horizontal axis and the intensity of amplitude in the verticalaxis to compute a MTF (modulation transfer function). The number oflines per pixel may be the number of lines per unit length at a distancefrom an actual position of the subject. Since this embodiment employsthe wedge-shaped radial chart, the directional MTF illustrated in FIG. 5may be analyzed.

FIG. 26 is a diagram illustrating a computational procedure of PSF(point spread function). In an example illustrated in FIG. 26, thedetermination part 102 initially computes angles and ellipticityaccording to positions of the image. The determination part 102 may beable to compute an ellipse by forming a contour line with apredetermined threshold (approximately half of the maximum amplitude) inthe MTF computed per predetermined angle.

The determination part 102 may be able to compute ellipticity based on alongitudinal axis and a short axis of the computed ellipse. Thedetermination part 102 geometrically computes an angle θ1 based on aposition of the image. The determination part 102 may also be able tocompute the angle θ1 based on the longitudinal axis and the short axisof the resolution ellipse. The latter method of computing the angle θ1based on the longitudinal axis and the short axis of the resolutionellipse may be able to compute the angle in compliance with the actualimage blurring. As already illustrated with reference to FIG. 3, theimage blurring in the radial direction may be large when a concentriccircle is drawn from the center of the optical axis.

The determination part 102 may, for example, compute an angle betweenthe vertical direction and the radial direction. Note that the center ofthe optical axis is basically the center of the image; however, thecenter of the image may be deviated (shifted) by the shift of the lensposition. The determination part 102 determines the PSF based on thecomputed ellipticity and angle. In this case, the ellipse of the PSFresides in a direction rotated by 90 degrees from the direction in whichthe ellipse formed of the contour line resides.

Referring back to FIG. 25, the determination part 102 determines, basedon the resolution analysis result of the corrected image, filter datahaving anisotropy of the inverse filter with respect to the correctedimage corrected by the filter (i.e., the above inverse filter) incompliance with the image blurring function (i.e., PSF).

Further, the determination part 102 determines the weighting factor(e.g., γ) by changing the weighting factor with respect to adifferential direction of the image. For example, the determination part102 determines the weighting factor of the Dx direction as 1, and theweighting factor of the Dx direction as γ, and then adjusts theweighting factor γ of the Dx direction. Accordingly, the resolutionanisotropy may be improved.

Further, the determination part 102 determines the weighting factor byrotating the direction (e.g., θ) with respect to the differentialdirection of the image. Accordingly, the direction of the degradedresolution may be detected and filtering may be performed based on thedetected direction of the degraded resolution.

Specifically, the determination part 102 may determine appropriateweighting factors ϵ and γ by, for example, adjusting the weightingfactors ϵ and γ. The weighting factor γ represents a weighting factor ofthe filter parameter in the direction of the degraded resolution. Thefilter parameter in the direction of the degraded resolution may, forexample, be a complex conjugate of the Dy (ω, θ) associated with theweighting factor γ of the formula (18) and its Dy (ω, θ).

Since the determination part 102 determines the weighting factor byadjusting the weighting factor, the determination part 102 includes anadjustment part 121, an image correcting part 122, a coefficientdetermination part 123, and a filter determination part 124.

The adjustment part 121 may, for example, adjust the weighting factor ϵthat does not depend on the directions and the weighting factor γ thatdepends on the directions. The adjustment part 121 sets initial valuesof the weighting factors 6 ϵ and γ, and transmits the initial values tothe image correcting part 122.

The image correcting part 122 performs the image correction based on theweighting factors acquired from the adjustment part 121. The imagecorrecting part 122 corrects the image by filtering the image utilizingthe inverse filter illustrated in the formula (18). The image correctingpart 122 transfers the corrected image to the resolution analysis part101 so as to analyze the deterioration of the resolution again.

The coefficient determination part 123 determines the weighting factorsbased on the resolution analysis result with respect to the correctedimage such that the difference in the deterioration of the resolutionbetween the two directions is small. The coefficient determination part123 retains the analysis results of the corrected images corrected basedon various weighting factors, and determines the weighting factors ϵ andγ such that the difference between the values of the spatial frequencyis minimized (a determination process 1).

Further, the coefficient determination part 123 may determine theweighting factors ϵ and γ such that the difference between theintensities of the amplitude in a predetermined spatial frequency isminimized (a determination process 2).

Note that plural thresholds may be set for each of the thresholds 1 and2. The coefficient determination part 123 may determine the weightingfactors such that the sum of squared difference in each of thethresholds is minimized. Note that the coefficient determination part123 may determine the weighting factors such that the predetermineddifference is less than or equal to the threshold set in advance. Thisthreshold may be set in the experiments performed in advance.

Further, the coefficient determination part 123 may determine theweighting factor such that the difference between the sum of squareddifference between two directional resolutions in the central part ofthe image and the sum of squared difference between two directionalresolutions in the peripheral part of the image differing from thecentral part of the image is a predetermined value or less. In addition,the coefficient determination part 123 may determine the weightingfactors such that the difference between the sum of squared differencebetween the central part and the peripheral part of the image isminimized

This is because when reducing the resolution anisotropy, the resolutionof the entire image may be uniform by equating the resolution of thecentral part of the image with the resolution of the peripheral part ofthe image, which may improve the image quality.

The determination in minimizing the weighting factors made by thecoefficient determination part 123 may be computed by utilizing aminimization function or may be judged by a person. Examples of theminimization function include a simplex search method (or simplexalgorithm), a steepest-descent method, and a conjugate gradient method.

The determination part 102 is configured to adjust the weighting factorby changing the weighting factor, compute an inverse filter based on theadjusted weighting factor, correct an image by the computed inversefilter, and determine an optimal weighting factor based on theresolution analysis result of the corrected image. The determinationpart 102 repeatedly adjusts the weighting factor, computes the inversefilter, corrects the image by the inverse filter, and analyzing theresolution of the image until determination part 102 determines theoptimal weighting factor.

The filter determination part 104 is configured to compute an inversefilter K_(inv) utilizing the optimal weighting factor determined by thecoefficient determination part 123, and then compute an inverse filterk_(inv) in a spatial domain based on the inverse filter K_(inv) in thefrequency domain. Note that the inverse filter k_(inv) in the spatialdomain is hereinafter called a “spatial filter”.

$\begin{matrix}{x = {k_{inv} \otimes y}} & (19) \\{k_{inv} = {f^{- 1}\frac{\overset{\_}{K\left( {\omega,\theta} \right)}}{\begin{matrix}{{{K\left( {\omega,\theta} \right)}\overset{\_}{K\left( {\omega,\theta} \right)}} +} \\{ɛ\left\{ {{{D_{X}\left( {\omega,\theta} \right)}\overset{\_}{D_{X}\left( {\omega,\theta} \right)}} + {{{\gamma D}_{Y}\left( {\omega,\theta} \right)}\overset{\_}{D_{Y}\left( {\omega,\theta} \right)}}} \right\}}\end{matrix}}}} & (20)\end{matrix}$Since the number of taps (elements) is extracted from the computedinverse filter k_(inv) the computed inverse filter k_(inv) is limited toa finite filter.

Note that the coefficient analysis part 10 performs the above processeswithin the image having a chart. The coefficient analysis part 10analyzes the resolution anisotropy at respective positions within theimage, and determines the spatial filter for improving such resolutionanisotropy.

The spatial filter for improving the resolution anisotropy whilecorrecting image blurring may be determined with respect topredetermined image positions by performing the aforementionedprocesses. For example, the coefficient analysis part 10 may detect thedirection having resolution degraded greater than the resolutions inother directions so as to determine the weighting factor to improve thedegraded resolution in the detected direction.

The coefficient analysis part 10 performs the spatial filter computationat respective positions of the image. FIG. 27 is a diagram illustratingan example of an image having 12 wedge charts. FIG. 27 illustrates onlyone example of the image having 12 wedge charts, and the image may havethe number of wedge charts other than 12 insofar as the charts reside inthe corresponding divided regions of the image.

The coefficient analysis part 10 determines filter data in respectiveregions having the charts to compute the spatial filters. Thecoefficient analysis part 10 creates a table having the spatial filterseach associated with the corresponding positions of the image.

FIG. 28 is a diagram illustrating an example of a spatial filter table.The spatial filter table illustrated in FIG. 28 associates upper leftpixel coordinates in the regions having respective charts of the image,and the corresponding spatial filters computed based on the computedcharts in the respective regions. For example, the position (x1, y1) isassociated with a spatial filter (FIL1) as illustrated in FIG. 28.

Accordingly, the image processing apparatus having the coefficientanalysis part 10 may be able to determine a spatial filter havingimproved resolution anisotropy. The spatial filter having the improvedresolution anisotropy is not limited to the above-described example. Thespatial filter having the improved resolution anisotropy may be thespatial filter computed by the technology disclosed in Patent Document 1(i.e., Japanese Laid-open Patent Publication No. 2012-23498). In such acase, the spatial filter may need to have resolution anisotropy.

Note that the image processor 4 may, for example, be formed of a(digital signal processor). In this case, the RAW memory 41 may be a DSPembedded memory or an external memory. Further, the post-processor 5,the image memory 8, the coefficient analysis part 10, the VRAM fordisplay and the like may integrally be formed as an integrated DSPtogether with the image processor 4. Alternatively, the coefficientanalysis part 10 may be formed as a single unit or may be formed as theimage processing apparatus including other processors.

Alternatively, the function of the image processor 4 or the coefficientanalysis part 10 may be realized not by a specifically designatedprocessor such as DSP but by causing a general-purpose processor such asa CPU (central processing unit) to execute a predetermined program.Similarly, the drive controller 6, the controller 7, and thepost-processor 5 may also be formed of at least one specificallydesignated processor or general-purpose processor.

Note that the program causing the processor to function as the imageprocessor 4 or the coefficient analysis part 10 and a recording mediumrecording such a program may also serve as embodiments of the presentinvention. The above recording medium has a non-transitoryconfiguration, and is not a transitory medium such as a signal itself.

Problem of Making Finite Spatial Filter

Next, a problem that may arise when making a finite spatial filter isdescribed. When making the finite spatial filter having anisotropy,information contained in parts other than the extracted taps (elements)may be lost (missing). Since the spatial filter has anisotropy, missinginformation may differ in different directions. In addition, a totalnumber of added elements may differ in the different directions.

Since the missing information contains high frequency information,levels of the correction (correcting degrees) may differ in differentcorrecting directions when correcting the image utilizing the finitespatial filter.

FIG. 29 is a diagram illustrating an example of a spatial filter. Thespatial filter illustrated in FIG. 29 includes anisotropy based on theabove-described formula (20). In FIG. 29, the elements of the spatialfilter are expressed by different colors.

FIG. 30 is a diagram illustrating the intensity of the finite spatialfilter. In the example illustrated in FIG. 30, when the number of tapsis nine, the number of elements corresponding to nine taps (i.e., 9×9)is extracted from the center of the spatial filter. As illustrated inFIG. 30, the vertical axis represents the intensity of the finitespatial filter. When a finite number of elements is extracted from thespatial filter, information is lost in parts indicated by a101. Further,since the spatial filter exhibits anisotropy, high frequency informationto be lost may differ in different directions.

Thus, levels of the correction (correcting degrees) may differ in thedifferent directions when using the finite spatial filter. Asillustrated in FIG. 31, when the chart is corrected by the spatialfilter having nine taps, a moiré pattern is generated. FIG. 31 is adiagram illustrating an example of a moiré pattern generated in acorrected image. As illustrated in FIG. 31, a moiré pattern is generatedby making the spatial filter finite.

FIG. 32 is a diagram illustrating a resolution analysis result of theimage corrected by the finite spatial filter. As illustrated in FIG. 32,correcting degrees of deterioration in the resolution differ indifferent directions.

FIG. 33 is a diagram illustrating a resolution analysis result of theimage corrected by suppressing the moiré pattern. In the exampleillustrated in FIG. 33, anisotropy remains in the resolution as a resultof suppressing the improvement of the high frequency for preventing themoiré pattern from being generated.

In the following, illustrations are given of embodiments of preventingthe generation of the moiré pattern when making the spatial filterfinite.

First Embodiment

Initially, an outline of preventing the generation of the moiré patternis described. FIG. 34 is a diagram illustrating drastic luminancechanges. In FIG. 34, parts (e.g., rectangular parts) illustrating thedrastic luminance changes include various high frequency components.

As described above, the moiré pattern is generated because differenthigh frequency information may be lost in different directions by makingthe spatial filter having resolution anisotropy finite. That is, themoiré pattern is generated since levels of the correction (i.e.,correcting degrees) of frequency deterioration differ in differentdirections.

Hence, the parts exhibiting drastic luminance changes may be reduced andthe difference in the levels of the correction (i.e., correctingdegrees) of the high frequency information between the differentdirections may be reduced by causing the image to pass through a finitehigh-pass filter. Accordingly, generation of the moiré pattern due tofrequency information deterioration in the different directions may beprevented.

FIG. 35 is a diagram illustrating luminance changes after the image ispassed through a high-pass filter. The example illustrated in FIG. 35illustrates a resultant state after the luminance changes illustrated inFIG. 34 are differentiated twice.

Note that the finite high-pass filter may be a finite filter having thesumb of elements being 0, and at least two of the elements being non-0.In the following, the finite high-pass filter may be described as anexample of the finite filter.

Next, an image pickup apparatus including the image processing apparatusaccording to the first embodiment is illustrated. In the firstembodiment, when making the finite spatial filter having resolutionanisotropy, the generation of the moiré pattern may be prevented byconvolving the finite high-pass filter.

Configuration

FIG. 36 is a block diagram illustrating a schematic configurationexample of an image pickup apparatus including the image processingapparatus according to the first embodiment. In the configurationillustrated in FIG. 36, elements similar to those of the configurationillustrated in FIG. 21 are provided with the same reference numerals. Inthe following, an image processor 15 in the first embodiment is mainlydescribed.

The image processor 15 includes a RAW memory 41, a filter controller151, and a filter processor 152. The filter controller 151 maintains thespatial filter table illustrated in FIG. 28. The filter controller 151is configured to convolve a finite high-pass filter with respect to eachof the spatial filters in the spatial filter table. The filtercontroller 151 outputs the spatial filter having the high-pass filterconvolved to the filter processor 152. That is, the filter controller151 outputs to the filter processor 152 each of the spatial filterscorresponding to the positions of the image to be processed.

The filter processor 152 executes filtering at the respective positionsof the image utilizing the finite spatial filters acquired from thefilter controller 151. This may improve the different resolutionanisotropy at the respective positions of the image to prevent thegeneration of the moiré pattern, which may result in improving the imagequality.

Filter Controller and Filter Processor

Next, the filter controller 151 and the filter processor 152 in thefirst embodiment are illustrated. FIG. 37 is a block diagramillustrating a schematic configuration example including the filtercontroller 151 and the filter processor 152 according to the firstembodiment. Initially, the filter controller 151 is illustrated. Thefilter controller 151 includes a filter storage 201, a filteracquisition part 202, and a filter calculation part 203.

The filter storage 201 is configured to store a first spatial filter211, and a second spatial filter 212. The respective filters may bestored in different storage regions of the filter storage 201. The firstspatial filter 211 serves as a spatial filter having resolutionanisotropy. The first spatial filter 211 may, for example, be each ofthe filters in the spatial filter table illustrated in FIG. 28. Thesecond spatial filter 212 serves as a filter computed by the filtercalculation part 203. The second spatial filter 212 may, for example, beobtained by convolving a high-pass filter with the first spatial filter211.

The filter acquisition part 202 is configured to acquire a finitespatial filter having image resolution anisotropy. The filteracquisition part 202 may, for example, acquire the first spatial filter211 from the filter storage 201. The filter acquisition part 202 outputsthe acquired first spatial filter 211 to the filter calculation part203.

The filter calculation part 203 is configured to compute the secondspatial filter 212 by convolving a finite filter having the sum ofelements being 0 and at least two of the elements being non-0 withrespect to the first spatial filter 211 that is acquired from the filteracquisition part 202. The filter calculation part 203 may, for example,compute the second spatial filter 212 by convolving a finite high-passfilter with the first spatial filter 211.

The filter calculation part 203 retains the finite high-pass filters inadvance. The finite high-pass filter may, for example in a case of a 3×3filter, be computed by the following formula (21) or (22).

$\begin{matrix}{{Lap} = \begin{bmatrix}0 & 1 & 0 \\1 & {- 4} & 1 \\0 & 1 & 0\end{bmatrix}} & (21) \\{{Lap} = \begin{bmatrix}1 & 1 & 1 \\1 & {- 8} & 1 \\1 & 1 & 1\end{bmatrix}} & (22)\end{matrix}$

Since the filter according to any angles of directions processes theanisotropy in the embodiment, it is preferable that all the elements ofthe filter each have non-0 coefficient.

When the high-pass filter is 3×3, the filter calculation part 203 may,for example, determine the spatial filter k_(inv) as 7×7 so as tocompute a 9×9 filter by convolving the two filters (i.e., the 3×3high-pass filter and the 7×7 spatial filter). As described above, thefilter calculation part 203 computes a desired filter (i.e., 9×9 filterin this case) having a desired number of taps by convolving thehigh-pass filter and the spatial filters.

Note that the 7×7 spatial filter is represented by F₇, and the high-passfilter is represented by Lap, and the 9×9 filter F9 computed by thefilter calculation part 203 may be represented by the following formula(23).F₉=Lap⊗F₇   (23)

FIG. 38 is a diagram illustrating intensity (a first example) of each ofpixels in the finite spatial filter F₉. The example illustrated in FIG.38 indicates the intensity change between the two directions (i.e., thehorizontal direction Dx, the vertical direction Dy) of the finitespatial filter. FIG. 39 is a diagram illustrating intensity (a secondexample) of each of pixels in the finite spatial filter F₉. In theexample illustrated in FIG. 39, the intensity of each of the pixels isexpressed by a different color. As illustrated in FIGS. 38 and 39, sincean extra amount of correction will not be applied to parts exhibitingdrastic luminance change, it may be possible to make stable correctionin the parts exhibiting the drastic luminance change.

The filter calculation part 203 stores the second spatial filter F₉computed by the aforementioned formula (23) into the filter storage 201.

Note that the filter calculation part 203 may be provided in a separateapparatus, and hence, the second spatial filter 212 computed by theseparate apparatus may be stored in the filter controller 151.

Referring back to FIG. 37, the filter controller 152 is illustrated. Thefilter controller 152 includes a convolution part 301, and a subtracter302. The convolution part 301 is configured to acquire an image from theRAW memory 41, and convolve the image by the second spatial filter F₉ toperform a filter process on the image. The convolution part 152 outputsthe filter-processed image to the subtracter 302.

The subtracter 302 subtracts the filter-processed image from the imageacquired from the RAW memory 41 to generate the corrected image. Notethat when x represents the corrected image, and y represent the originalimage, the following formula (24) is established.x=y−F₉⊗y  (24)

FIG. 40 is a diagram illustrating an image (a concept) of a correctingprocess according to the first embodiment. As illustrated in FIG. 40,the filter processor 152 performs a deconvolution filter process on theoriginal image, and generates the corrected image by subtracting thefilter-processed image from the original image. The original image isacquired from the RAW memory 41, and the deconvolution filter is thefinite spatial filter F₉.

Note that the filter processor 152 may be configured not to utilize onespatial filter corresponding to each pixel in each of the regions of theimage, but may instead perform linear interpolation utilizingneighboring spatial filters to compute a pixel value of the focused-onpixel.

FIG. 41 is a diagram illustrating the linear interpolation of thefocused-on pixel. As illustrated in FIG. 41, the filter processor 152may, for example, perform linear interpolation according to a distanceof each of the pixels by utilizing the central pixel of each of theregions computed by four neighboring spatial filters so as to compute apixel value of the focused-on pixel. In the example illustrated in FIG.41, the filter processor 152 linearly interpolates the pixel values ofthe regions respectively computed by FIL1, FIL2, FIL5, and FIL6 tocompute the pixel value of the focused-on pixel.

Alternatively, the filter processor 152 may compute the pixel value ofthe focused-on pixel after computing the spatial filters themselves withrespect to the focused-on pixel by performing linear interpolation.Further, in the above example, the number of neighboring spatial filtersis four; however, the number of neighboring spatial filters is notlimited to four. The number of neighboring spatial filters may be anynumber other than four. In addition, in the above example, the distanceis subjected to linear interpolation; however, any other interpolationmethods may be performed. Moreover, the interpolation may be performedon each of the further segmented regions or each of the pixels.

Effect

Next, an advantageous effect of employing the above-described spatialfilter F₉ is described. FIG. 42 is a diagram illustrating a resolutionanalysis result before the image correction. As illustrated in FIG. 42,the resolution in the Dy direction exhibits deterioration greater thanthe resolution in the Dx direction.

FIG. 43 is a diagram illustrating a resolution analysis result after theimage correction according to the first embodiment. The exampleillustrated in FIG. 43 employs the finite spatial filter F₉. Asillustrated in FIG. 43, the resolutions in the Dx and Dy directions areboth improved, and the difference in the resolution between the Dx andDy directions is decreased. Accordingly, FIG. 43 illustrates that theresolution anisotropy is improved. Further, the resolution analysisresult in FIG. 43 illustrates that the generation of the moiré patternis prevented by the reduction in the drastic luminance change.

Operations

Next, operations of the image processor 15 according to the firstembodiment are illustrated. FIG. 44 is a flowchart illustrating anexample of a filter computation process according to the firstembodiment.

In step S101, the filter acquisition part 202 acquires the first spatialfilter 211 from the filter storage 201.

In step S102, the filter calculation part 203 convolves a finitehigh-pass filter with the first spatial filter 211.

In step S103, the filter acquisition part 151 determines whether all thefilters of the first spatial filter 211 stored in the filter storage 201have been processed. When all the filters of the first spatial filter211 have been processed (“YES” in step S103), the filter computationprocess is ended, whereas when all the filters of the first spatialfilter 211 have not been processed (“NO” in step S103), step S101 isprocessed (back to step S101).

Accordingly, the second spatial filter 212 may be generated byconvolving the high-pass filter with the first spatial filter 211 atrespective positions of the image.

FIG. 45 is a flowchart illustrating an example of a filter processaccording to the first embodiment. In step S201 illustrated in FIG. 45,the convolution part 301 determines the second spatial filter accordingto a position of a region of the image to be processed.

In step S202, the convolution part 301 computes pixels of the regions ofthe image to be processed utilizing the determined filter. At thismoment, the filter processor 152 computes a pixel value, after thefilter process has been performed, at a focused-on position within theregion by performing linear interpolation.

In step S203, the convolution part 301 determines whether the correctionhas been performed on all the pixels. When the correction has beenperformed on all the pixels (“YES” in step S203), step S204 isprocessed, whereas when the correction has been performed not on all thepixels (“NO” in step S203), step S201 is processed (back to step S201).

In step S204, the subtracter 302 subtracts a filter-processed image fromthe original image to generate a corrected image.

Accordingly, the corrected image may be generated by preventing thegeneration of the moiré pattern while improving the resolutionanisotropy.

Second Embodiment

Next, an image pickup apparatus having an image processing apparatusaccording to a second embodiment is illustrated. In the secondembodiment, the filter process is simplified despite exhibiting anadvantageous effect similar to that obtained in the first embodiment.

Configuration

The image pickup apparatus according to the second embodiment is givenwith the same reference numeral as that given to the image pickupapparatus according to the first embodiment since the image pickupapparatus according to the second embodiment is similar to the imagepickup apparatus according to the first embodiment.

Filter Controller and Filter Processor

Next, a filter controller 151 and a filter processor 152 in the secondembodiment are illustrated. FIG. 46 is a block diagram illustrating aschematic configuration example including the filter controller 151 andthe filter processor 152 according to the second embodiment. Initially,the filter controller 151 is illustrated. The filter controller 151includes a filter storage 401, a filter acquisition part 202, and afilter calculation part 402.

In configuration illustrated in the second embodiment, elements similarto those of the configuration illustrated in the first embodiment areprovided with the same reference numerals.

The filter storage 401 is configured to store a third spatial filter 411computed by the filter calculation part 402.

The filter calculation part 402 is configured to compute the thirdspatial filter 411 that omits a subtraction process between the imagesin the first embodiment. The subtraction process may be omitted bymodifying the formula as follows.

Note that in the following formula (25), a filter I₀ having the centerrepresenting “1” is employed.

$\begin{matrix}{I_{0} = \begin{bmatrix}0 & \ldots & 0 \\\vdots & 1 & \vdots \\0 & \ldots & 0\end{bmatrix}} & (25)\end{matrix}$

When the formula (24) is modified with the formula (25), the followingformula (26) is obtained.x=I₀⊗y−F₉⊗y=(I₀−F₉)⊗y  (26)

Next, F′₉ is defined by the following formula (27).F′₉=I₀−F₉   (27)

When the formula (26) is modified with the formula (27), the followingformula (28) is obtained.x=F′₉⊗y   (28)

Accordingly, the result similar to that of the first embodiment may beobtained, thereby generating a finite spatial filter that omits thesubtraction process between the images.

The filter calculation part 402 is configured to compute the spatialfilter F′₉ represented by the formula (27). The filter calculation part402 is further configured to write the computed spatial filter F′₉ inthe filter storage 401. The spatial filter F′₉ is the third spatialfilter 411.

FIG. 47 is a diagram illustrating intensity of each of pixels in thefinite spatial filter F′₉. In the example illustrated in FIG. 47, theintensity of each of the pixels is expressed by a different color.

Referring back to FIG. 46, the filter controller 152 in the secondembodiment is illustrated. The filter controller 152 includes aconvolution part 501. The convolution part 501 performs the processindicated by the formula (28) to generate the corrected image x.

FIG. 48 is a diagram illustrating an image (a concept) of a correctingprocess according to the second embodiment. As illustrated in FIG. 48,the filter processor 152 performs a deconvolution filter process on theoriginal image to generate the corrected image. The original image isacquired from the RAW memory 41, and the deconvolution filter is thefinite spatial filter F′₉.

Note that the filter processor 152 may be configured not to utilize onespatial filter corresponding to each pixel in each of the regions of theimage, but may instead perform linear interpolation utilizingneighboring spatial filters to compute a pixel value of the focused-onpixel in a manner similar to the first embodiment.

Effect

The second embodiment omits the subtracting process between the imagesand simply modifies the formula. Hence, an advantageous effect similarto that of the first embodiment may be obtained.

Operations

Next, operations of the image processor 15 according to the secondembodiment are illustrated. FIG. 49 is a flowchart illustrating anexample of a filter computation process according to the secondembodiment.

In step S301, the filter acquisition part 202 acquires a first spatialfilter 211 from the filter storage 401.

In step S302, the filter calculation part 402 convolves a finitehigh-pass filter with the first spatial filter 211.

In step S303, the filter calculation part 402 computes the third spatialfilter F′₉ based on the formula (27).

In step S304, the filter acquisition part 151 determines whether all thefilters of the first spatial filter 211 stored in the filter storage 401have been processed. When all the filters of the first spatial filter211 have been processed (“YES” in step S304), the filter computationprocess is ended, whereas when all the filters of the first spatialfilter 211 have not been processed (“NO” in step S304), step S301 isprocessed (back to step S301).

Accordingly, the third spatial filter F′₉ may be generated by convolvingthe high-pass filter with the first spatial filter 211 at respectivepositions of the image and then subtracting the convolved filter fromthe filter I₀.

FIG. 50 is a flowchart illustrating an example of a filter processaccording to the second embodiment. In step S401 illustrated in FIG. 50,the convolution part 501 determines the third spatial filter accordingto a position of a region of the image to be processed.

In step S402, the convolution part 501 computes pixels of the region ofthe image to be processed utilizing the determined filter. At thismoment, the filter processor 152 computes a pixel value, after thefilter process has been performed, at a focused-on position within theregion by performing linear interpolation.

In step S403, the convolution part 501 determines whether the correctionhas been performed on all the pixels. When the correction has beenperformed on all the pixels (“YES” in step S403), the filter process isended, whereas when the correction has been performed not on all thepixels (“NO” in step S403), step S401 is processed (back to step S401).

Accordingly, the corrected image may be generated by preventing thegeneration of the moiré pattern while improving the resolutionanisotropy in a manner similar to the first embodiment.

Third Embodiment

Next, an image processing apparatus according to a third embodiment isdescribed. In the third embodiment, a CPU (central processing unit), aDSP (digital signal processor), or the like may perform respectiveprocesses of the aforementioned coefficient analysis part 10 and theimage processor 15 in the first or the second embodiment

Configuration

FIG. 51 is a block diagram illustrating a schematic configurationexample of the image processing apparatus according to the thirdembodiment. The image processing apparatus illustrated in FIG. 51includes a controller 602, a main storage 604, an auxiliary storage 606,a communication part 608, a recording medium IF part 610, and a cameraIF part 612. The above parts of the image processing apparatus accordingto the third embodiment are connected with one another via a bus suchthat they are configured to mutually transmit or receive data with oneanother via the bus. The image processing apparatus according to thethird embodiment may be an apparatus having an information processingfunction such as a PC (personal computer) or a server, a smartphone, ora tablet terminal.

The controller 602 may serve as a CPU (central processing unit), or thelike configured to perform controls of the various devices or operationsor manipulation of data within a computer. The controller 602 may alsoserve as an operating unit configured to execute programs stored in themain storage 604 or the auxiliary storage 606.

The controller 602 may, for example, implement the above-describedprocesses by executing a program for performing a coefficient analysisprocess or an image process stored in the auxiliary storage 606.

The main storage 604 may, for example, be a ROM (read only memory) and aRAM (random access memory) serving as a storage device configured tostore or temporarily store programs to be executed by the controller 602such as an OS (operating system) that is basic software and applicationsoftware.

The auxiliary storage 606 may be a HDD (hard disk drive), or the likeserving as a storage device configured to store data associated with theapplication software, and the like. The auxiliary storage 606 may alsostore programs for performing the coefficient analysis process or theimage process acquired from a recording medium 614.

The communication part 608 may perform wired or wireless communications.The communication part 608 may, for example, acquire plural images froma server, and store the acquired images in the auxiliary storage 608, orthe like.

A recording medium IF (interface) part 610 is configured to serve as aninterface between a recording medium 614 (e.g., a flash memory)connected via a transmission channel such as a USB (universal serialbus) and the image processing apparatus.

Further, a predetermined program stored in the recording medium 614 maybe installed in the image processing apparatus via the recording mediumIF part 610. The predetermined installed program may be executable bythe image processing apparatus.

The camera IF part 612 is configured to serve an interface forperforming communications with a camera 616. The camera IF part 612acquires from the camera 616 the chart images captured by the camera 616or normal images subject to correction and stores the acquired images inthe auxiliary storage 606.

The camera 616 is configured to capture the chart images illustrated inFIG. 6, or normal landscapes or individuals. The captured images areincorporated into the image processing apparatus via the camera IF part612. Note that the camera 616 may be embedded in the image processingapparatus.

Hence, the image processing apparatus may be able to acquire the chartimage to compute spatial filters at respective positions, and correctthe image subject by utilizing the computed spatial filters.

Accordingly, programs for implementing the above-described coefficientanalysis process and the image process may be stored in the recordingmedium, which may cause a computer to implement the above-describedcoefficient analysis process and image process.

The above programs may, for example, be recorded on the recordingmedium, allowing the computer, a mobile terminal, a smartphone, or atablet terminal to read the programs from the recording medium, therebyimplementing the above-described coefficient analysis process and imageprocess.

Note that various types of recording media may be used as the aboverecording medium, examples of which include a recording mediumoptically, electronically or magnetically recording information such asa CR-ROM (compact disk read-only memory), a flexible disk, and amagneto-optical disk, or semiconductor memory or the like electricallyrecording information such as a ROM (read-only memory), a flash memoryor the like. Such media do not include non-transitory media such as acarrier wave.

Further, charts illustrated in FIGS. 52 to 54 may be employed asexamples of the chart. FIG. 52 is a diagram illustrating an example ofthe chart (a first example). The resolution of the chart illustrated inFIG. 52 may be analyzed at least in two directions.

FIG. 53 is a diagram illustrating another example of the chart (a secondexample). The resolution of the chart illustrated in FIG. 53 may beanalyzed in all radial directions. The chart illustrated in FIG. 53 issuitable for the resolution analysis.

FIG. 54 is a diagram illustrating another example of the chart (a thirdexample). The chart illustrated in FIG. 54 has a difference in theresolution between its central part and its peripheral part. The chartillustrated in FIG. 53 (i.e., the second example) may be suitable forthe resolution analysis according to the embodiments; however, theanalysis of the resolution anisotropy may be conducted on the chartsillustrated in FIG. 52 (i.e., the first example) and in FIG. 54 (i.e.,the third example)

According to the technology disclosed above, the generation of the moirépattern may be prevented when a spatial filter having resolutionanisotropy is limited to a finite spatial filter.

As described above, the examples and embodiments of the image processingapparatus, the image pickup apparatus, the information processingmethod, and the non-transitory recording medium storing a program forperforming the information processing method have been described indetail; however, it should not be construed that the present inventionis limited to those specific examples and embodiments described above.Various changes or alternations may be made within the scope of theinvention. Further, all or parts of the elements recited in theembodiments may be combined.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority orinferiority of the invention. Although the embodiments of the presentinvention have been described in detail, it should be understood thatthe various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the invention.

What is claimed is:
 1. An image processing apparatus, comprising: anafirst acquisition unit configured to acquire a first finite spatialfilter having image resolution anisotropyby making a spatial filterfinite, so that the first finite spatial filter is limited to a finitenumber of elements less than or equal to an element number of thespatial filter being calculated from an inverse function of a functionrepresenting image blurring and used to improve anisotropy of aresolution of an image in which an imagethe resolution varies as afunction of separation and direction relative to a center of an image;and a calculationa second acquisition unit configured to computeacquirea second spatial filter by convolving a finite high-pass filter withrespect to the first finite spatial filter, the finite high-pass filterhaving a sum of elements being 0 and at least two of the elements beingnon-0, the second spatial filter being smaller than the first finitespatial filter and the finite high-pass filterimproving the resolutiondiffering in different directions and reducing a moiré pattern for thefirst finite spatial filter, the moiré pattern being generated by acorrection causing deterioration in high frequency components in thedifferent directions.
 2. The image processing apparatus as claimed inclaim 1, wherein the finite high-pass filter is a high-pass filterhaving all the elements being non-0 coefficients.
 3. The imageprocessing apparatus as claimed in claim 1, wherein the calculation unitcomputes a third spatial filter by subtracting the second spatial filterfrom a filter having a central element being 1 and elements other thanthe central element being
 0. 4. The image processing apparatus asclaimed in claim 1, further comprising: a filter processor configured toperform a filter process on an input image utilizing the second spatialfilter and subtract a filter-processed image from the input image togenerate a corrected image.
 5. The image processing apparatus as claimedin claim 3, further comprising: a filter processor configured to performa filter process on an input image utilizing the second spatial filterto generate a corrected image.
 6. An image pickup apparatus, comprising:the image processing apparatus as claimed in claim 1; and an imagepickup unit.
 7. A non-transitory computer-readable storage medium thatstores a program which, when executed by a processor, causes a computerto perform a process comprising: acquiring a first finite spatial filterhaving image resolution anisotropy by making a spatial filter finite, sothat the first finite spatial filter is limited to a finite number ofelements less than or equal to an element number of the spatial filterbeing calculated from an inverse function of a function representingimage blurring and used to improve anisotropy of a resolution of animage in which an image the resolution varies as a function ofseparation and direction relative to a center of an image; andcomputingacquiring a second spatial filter by convolving a finitehigh-pass filter with respect to the first finite spatial filter, thefinite high-pass filter having a sum of elements being 0 and at leasttwo of the elements being non-0, the second spatial filter being smallerthan the first finite spatial filter and the finite high-pass filterimproving the resolution differing in different directions and reducinga moiré pattern for the first finite spatial filter, the moiré patternbeing generated by a correction causing deterioration in high frequencycomponents in the different directions.
 8. An information processingmethod executed by a computer, the information processing methodcomprising: acquiring a first finite spatial filter having imageresolution anisotropy by making a spatial filter finite, so that thefirst finite spatial filter is limited to a finite number of elementsless than or equal to an element number of the spatial filter beingcalculated from an inverse function of a function representing imageblurring and used to improve anisotropy of a resolution of an image inwhich an image the resolution varies as a function of separation anddirection relative to a center of an image; and computingacquiring asecond spatial filter by convolving a finite high-pass filter withrespect to the first finite spatial filter, the finite high-pass filterhaving a sum of elements being 0 and at least two of the elements beingnon-0, the second spatial filter being smaller than the first finitespatial filter and the finite high-pass filterimproving the resolutiondiffering in different directions and reducing a moiré pattern for thefirst finite spatial filter, the moiré pattern being generated by acorrection causing deterioration in high frequency components in thedifferent directions.